A Prototype System for 3D Color Fusion and Mining of Multisensor / Spectral Imagery

We have developed a prototype system in which a user can fuse up to 4 modalities (or 4 spectral bands) of imagery previously registered to one another with respect to a 3D terrain model. The color fused imagery can be draped onto the terrain to support interactive 3D flythrough. The fused imagery, and its opponent-sensor contrasts, can be further processed to yield extended boundary contours and texture measures. Together, these layers of registered imagery and image features can be interactively mined for objects of interest. Data mining for infrastructure and compact targets is achieved using a point-and-click user interface in conjunction with a Fuzzy ARTMAP neural network for on-line pattern learning and recognition. Graphical user interfaces enable the user to control each stage of processing: image enhancement, image fusion, contour and texture extraction, 3D terrain characterization, 3D graphics model building, preparation for exploitation, and interactive data mining. The system is configured as a client-server architecture, enabling remote collaborative exploitation of multisensor imagery. Throughout, the processing of imagery and patterns relies on neural network models of spatial and color opponency, and the adaptive resonance theory of pattern processing. This system has been used to process imagery of a variety of geographic sites, in order to extract roads, rivers, forests and orchards, and performance has been assessed against manually determined “ground truth.” The data mining approach has been extended for the case of hyperspectral imagery of hundreds of bands. This prototype system has now been installed at multiple US government sites for evaluation by image analysts. We plan to extend this approach to include various nonimaging sensor modalities that can be localized to geographic coordinates (e.g., GMTI and SIGINT). We also plan to embed these image fusion and mining capabilities in commercial open software environments for image processing and GIS.